WE CLAIM; 



1. An interactive information processing system 
responsive*, to human operator movement of physical objects 
comprising: 

a discriminator for discriminating physical 
objects and generating an electronic physical object image 
signal representative of physical attributes of a physical 
ob j ect ; and 

a menu driven physical object attribute 
definition system for accommodating operator storage of 
physical attributes of a plurality of physical objects 
utilizing the electronic physical object image signal from 
the discriminator, 

2. A system according to claim 1, wherein a 
plurality of different types of physical objects are 
provided which correspond to physical objects normally used 
in handling paper files in an office, and wherein the 
attribute definition system includes means for 
accommodating operator indication of the type of physical 
object represented by the physical object image signal. 

3. A system according to claim 2, wherein said 
physical objects include objects representative of at least 
two of the following types of physical objects: 

(a) file of papers; 

(b) mail tray; 
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(c) fax machine; 

(d) computer keyboard ; 

(e) computer mouse; 

( f ) pen ; 

5 (g) memo; 

(h) marker pen; 

(i) eraser; 

(j) operator's hands in an open release 
position; and 

10 (k) operator's hands in a closed condition. 



W 4. A system according to claim 2, wherein sa.id types 

|fl of physical objects include the operator's hands in open 

. £~ 

y and closed condition, and the following other types of 

physical objects: 
iri 15 (a) file of paper; 

(b) pen; 

(c) eraser; 

(d) mail tray; 

(e) . fax machine; 
2 0 (f) keyboard; and 

(g) mouse. 



5. An information processing system according to 
claim 1, comprising: 

a file storage unit containing electronically 
25 stored information to be processed, including information 
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from the physical attribute definition system reflective of 
the physical objects to be used with the system; 

a link storage unit containing information 
reflective of a physical object being concurrently 
discriminated by the discriminator; and 

a file operator for processing information from 
the file storage unit in response to operator movement of 
at least one of the physical objects with respect to an 
image display reflective of at least one other of the 
physical objects. 

6. An information processing system according to 
claim 2, comprising 

a file storage unit containing electronically 
stored information to be processed, including information 
from the physical attribute definition system reflective of 
the physical objects to be used with the system; 

a link storage unit containing information 
reflective of a physical object being concurrently 
discriminated by the discriminator; and 

a file operator for processing information from 
the file storage unit in response to operator movement of 
at least one of the physical objects with respect to an 
image display reflective of at least one other of the 
physical objects. 

7. An information processing system according to 
claim 2, comprising a display surface in the form of a desk 



top, said physical objects being selectively placed on the 
display surface by the operator. 



8. An information processing system according to 
claim 1 , wherein said discriminator is a camera which 

5 continuously views the display surface during information 

processing operations. 

9. An information processing system according to 
claim 1 , wherein said attribution definition system 

Q includes an operator selectable menu at the desk top which 

ffl 10 menu includes operator selection areas for operator 

\^ ' 

ijl designation of a plurality of types of physical objects to 

y be used by the operator, whereby the operator can 

sequentially place the physical objects to be used on the 
In display surface and store their physical attributes, 

15 preliminarily to conducting actual information processing 

operations using the physical objects. 

10. An information processing system according to 
claim 5, wherein said file storage unit includes 
information reflective of predetermined distances between 

20 respective ones of the physical objects and image display 

reflection of the physical objects, and wherein said file 
operator includes means for initiating predetermined 
processing steps in response to movement of a physical 
object and an image display to positions within and without 

25 said predetermined distances. 
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11. An information processing system according to 
claim 10, wherein said information reflective of 
predetermined distances is indicative of characteristic 
point coordinates on said respective physical objects and 

5 image displays. 

12. An information processing system according to 
claim 11, wherein said characteristic point coordinates 
correspond to predetermined geometric locations on said 
physical objects and said image displays. 

m 10 13- An information processing system according to 

iji claim 11, wherein said characteristic point coordinates 

y includes a point on a hand of the operator. 

|S 14 . An information processing system according to 

%i claim 5, wherein said file operator includes hand gesture 

1=7 15 recognizing means for recognizing operator hand gestures 

reflecting release and hold operations for associated 

electronic files to be manipulated. 

15. An information processing system according to 
claim 14, wherein said hand gesture recognizing means 
2 0 recognizes a closed operator hand as a hold gesture and a 

hand gesture with open fingers as a release gesture. 

16 . An information processing system according to 
claim 14, wherein said hand gesture recognizing means 
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recognizes the release and hold gestures by the size of the 
image reflective of the distance of the hand from a camera 
of the discriminator. 

17, An information processing apparatus comprising: 
a display unit; 

a camera unit for photographing an image object 
displayed on said display unit, and a physical object and 
the hands of an operator placed on said display unit; 

recognition means for recognizing the physical 
object from an image from said camera unit; 

attribute definition means for defining an 
associated motion between the physical object recognized by 
said recognition means and said image object; 

attribute association processing means for 
executing an associated motion defined by said attribute 
definition means for said image object when a comparison 
value between a characteristic point coordinate of the 
physical object recognized by said recognition means and 
another characteristic point coordinate of another physical 
object, or between characteristic point coordinates in said 
image object is within a predetermined value; and 

means for automatically starting said attribute 
association processing means when said image object or said 
physical object is recognized to be a hold-state by gesture 
recognition means . 



18 . An information processing apparatus provided with 
a display unit with a display surface on a horizontal 
arrangement in which data can be displayed in a document 
sheet format, and a camera unit for photographing the hands 
of an operator placed on the display surface of said 
display unit as well as the data displayed on said display 
surface of said display unit, comprising: 

definition file store means for storing various 
information processing motions each corresponding to each 
of various states of images representative of hand 
gestures; and 

process execution means for executing a 
particular information processing operation for said data 
upon recognition that a distance between an image of the 
hand and said data is judged to be within a predetermined 
value according to images from said camera unit, and that 
an association for carrying out said particular information 
process operation intended by the operator is identified 
between the image of the hand and said definition file 
store means . 

19. The information processing apparatus according to 
claim 18, wherein a set of information processing 
operations corresponding to an image of a physical object 
which is physically corresponding to an image of a physical 
object which is physically placed on said display surface 
are stored in said definition file store means, and wherein 
said process execution means is caused to execute a 
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designated information processing operation corresponding 
to the physical object with respect to said data when said 
data is moved through manipulation of the image of the hand 
toward the physical object on the display surface as much 
as to come into a predetermined distance. 

20. The information processing apparatus according to 
claim 19, wherein an icon instead of said physical object 
displayed on said display unit is used. 

21. The information processing apparatus according to 
claim 19, wherein s\id data moved within said predetermined 
distance is highlighted . 

22. The informatiori processing apparatus according to 
claim 20, wherein said datV moved within said predetermined 
distance is highlighted. \ 

.23, An information proc/es^ing apparatus provided with 
a flat panel display capable>d^ displaying a document 
object including plural pagesX each being partially 
overlapped, comprising: \ 

layout coordinate designation means for 
designating a start point coordinateX and an end point 
coordinate for layout ing said document ocnect; 

page layout coordinate calculation means for 
obtaining respective coordinates for layout\ng respective 



-81- 



pages of said aocument from said start and said end point 
coordinates ; ano 

layoutX display means for displaying respective 
pages of said document on respective coordinates obtained 
by said page layouA coordinate calculation means. 
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24 . The information processing apparatus according to 
claim 23, wherein a layout coordinate point obtained by 
said page layout coordinate calculation means is an 
interpolated coordinate \ point which is obtained by 
interpolating a start poin\- coordinate, a coordinate, and 
between said start point and. said end point coordinates 



15 



25. The information processing apparatus according to 
claim 18, further comprising: tp^an^ for simultaneously and 
dynamically displaying a chr^e ^ which links between 
respective layout coordinates of~t^e ^document object during 
layout operation by said layout \ coordinate designation 
means, and said document object as well. 



26. The information processing apparatus according to 
claim 23, wherein said layout coordinaue designation means 

20 monitors and detects characteristic paints of the both 

hands of the operator by means of a monitor camera thereby 
to designate respective coordinates. 

27. A method of information processing using an 
electronic computing system and a plurality \of operator 
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manipulateci physical objects corresponding to traditionally 
used objecup for handling paper files in an office, 
comprising ; 

seleY;ting a plurality of physical objects; 
5 elect^ronically imaging a first one of the 

physical objects, and generating a first electronic image 
signal ref lectivA of the first physical object, utilizing 
an object attribute definition system menu to manually 
classify the first rahysical object by object type and to 
10 store the first electronic image signal with its object 

type; 

electronically imaging a second one of the 
physical objects and generating a second electronic image 
signal reflective of the \second physical object, utilizing 

15 the object attribute depainition system menu to manually 

classify the second physical object type and to store the 
second electronic image sigrml with its object type; and 

subsequently processing information in a file 
storage unit of the electronic^ computing system by moving 

20 at least one of the physical objects with respect to an 

image display reflection of at\ least one other of the 
physical objects 



28. A method of informatior\ processing using an 
electronic computing system according to claim 27, wherein 
25 said physical objects include the operator's hands in open 

and closed conditions and at least one of the following 
types of physical objects: 
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\![a) file of papers; 
(b) mail tray; 
(c\ fax machine; 

(d) \ computer keyboard; 

(e) Computer mouse; 

(f) P^n; 

(g) meXrio; 

(h) marlfier pen; 

(i) eraser; and 

comprisingX electronically imaging and manually 
classifying each of tfte physical objects and storing their 
images along with theiA selected object type. 



29, An informationX processing system responsive to 
user movement of a physical object, comprising: 
15 a display unit, 

a physical objec/fc discriminator, 

a file storage Lnii\\ containing electronically 
stored information, 

a link storage unit containing information 
20 reflective of a physical objeat discriminated by the 

discriminator and corresponding iJ^formation in the file 
storage unit, 

and a file operator for processing information 
from the storage unit and link storage unit to accomplish 
25 , interactive processing of information ^rom the file storage 
unit in response to at least one characteristic of the 
physical object. 
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3 0. \ A system according to Claim 29, wherein said 
discrimina\:or is operable to discriminate user hand 
movements , 

31. A )pystem according to Claim 30, wherein said 
discriminator operable to discriminate a plurality of 
different hand movements and to provide the link storage 
unit with a corresponding plurality of discrete signals 
which are processed by the operator to display respective 
different displays .\ 

10 32. A system aVicording to Claim 29, wherein said 

discriminator includes a camera. 



33. A system according to Claim 29, wherein said file 
operator is operable to/ Vetrieve electronic files in 
response to placement of a^b^csAcal object in a position to 
p 15 be discriminated by the discriminator. 



34. An information processing system using a user 
manipulated object, comprising: 
a display device, 

file storage means for\ storing electronic 
2 0 information, 

discrimination means for discriminating an object 
indicated by a user, 

link storage means for storing link information 
which designates the correspondence betweeji said object and 
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the electronic information stored in said file storage 
means , and 

file Operation means for reading the electronic 
information corresponding to the discrimination result of 
the object by said discrimination means from said file 
storage means on tWe basis of said link information and for 
displaying it on sai)d display device. 



m 



35, An information processing system using a user 
manipulated object, comprising: 
10 a display device, 

file storage mpans for storing electronic 
information, 

discrimination meana^or discriminating an object 
indicated by a user, 
15 link storage means fo^ storing link information 

which designates the correspondenoe between said object and 
the electronic information store(\ in said file storage 
means , 

link definition means hpr generating link 
20 information for displaying the elebtronic information 

corresponding to the discrimination result of the object on 
said display device, 

means for displaying the discriiMnation result of 
the object by said discrimination means ana the electronic 
25 information stored in said file storage means, and 
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means for designating the correspondence between 
the selected Electronic information among said displayed 
electronic infoVniation and said discrimination result. 

36. An information processing system according to 
Claim 35, wherein the discrimination result by said 
discrimination mea\is includes at least information 
indicating whether the indicated object identification 
number and said objeco are indicated or not. 
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37, An informationX processing system using an object 
according to Claim 35, wherein said link definition means 
operates when the link information for designating the 
correspondence between said Indicated object discrimination 
result and electronic infornjation is not stored in said 
link storage means. 



2 15 38. An information procei^sing system using a user 

manipulated object, comprising: 
a display device, 

file storage means f^r storing electronic 
inf ormat ion , 

20 discrimination means for dd^scriminating an object 

indicated by a user, 

link storage means for storing link information 
which designates the correspondence between said object and 
the electronic information stored in ipaid file storage 

2 5 means, and 
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link definition means for displaying the 
electronic imformation corresponding to the discrimination 
result of theXobject on said display device and displaying 
a dialogue screen for accepting a change in said stored 
link information on said display device. 

39. An infoAnation processing system using an object 
wherein said system comprises : 
a display \ievice , 

file stora^ge means for storing electronic 
10 information, 

discrimination means for discriminating an object 
indicated by a user, 

link storage me^ns for storing link information 
which designates the correspondence between said object and 
15 the electronic information/ \std|pred in said file storage 

means , 

attribute discrimination means for discriminating 
the attribute information added to said object, and 

file operation means foA executing the operation 
20 on the basis of said discriminated attribute information 

for the electronic information havmig the correspondence 
with said discriminated object 



25 



40. An information processing syst^em using an object 
according to Claim 39, wherein the attribute information 
added to said object incudes at least Ynformation for 
designating whether the electronic \ information 
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corresponding to ^j*!^ object i 



allowed to be modified or 



1x1 
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